Visualizing Data: plotly express

There are many ways to interact with data, and one of the most powerful modes of interaction is through visualizations. Visualizations show data graphically, and are useful for exploring, analyzing, and presenting datasets. We use four libraries for making visualizations: pandas, Matplotlib, plotly express, and seaborn. In this section, we'll focus on using plotly express.

Scatter Plots¶

A scatter plot is a graph that uses dots to represent values for two different numeric variables. The position of each dot on the horizontal and vertical axis indicates values for an individual data point. Scatter plots are used to observe relationships between variables, and are especially useful if you're looking for correlations.

After cleaning the data, we can use plotly express to draw scatter plots by specifying the DataFrame and the interested column names.

Practice

Plot the scatter plot for column "price" and "surface_total_in_m2".

3D Scatter Plots¶

Scatter plots can summarize information in a DataFrame. Three dimensional scatter plots look great, but be careful: it can be difficult for people who might not be sure what they're looking at to accurately determine values of points in the plot. Still, scatter plots are useful for displaying relationships between three quantities that would be more difficult to observe in a two dimensional plot.

Let's take a look at the first 50 rows of the mexico-city-real-estate-1.csv dataset.

Notice that the plot is interactive: you can rotate it zoom in or out. These kinds of plots also makes outliers easier to find; here, we can see that houses have higher prices than other types of properties.

Practice

Modify the DataFrame to include columns for the base 10 log of price and surface_covered_in_m2 and then plot these for the entire mexico-city-real-estate-1.csv dataset.

Mapbox Scatter Plots¶

A mapbox scatter plot is a special kind of scatter plot that allows you to create scatter plots in two dimensions and then superimpose them on top of a map. Our mexico-city-real-estate-1.csv dataset is a good place to start, because it includes location data. After importing the dataset and removing rows with missing data, split the lat-lon column into two separate columns: one for latitude and the other for longitude. Then use these to make a mapbox plot. Unfortunately, at present this type of plot does not easily allow for marker shape to vary based on a column of the DataFrame.

Practice

Create another column in the DataFrame with a log scale of the prices. Then create three separate plots, one for stores, another for houses, and a final one for apartments. Color the points in the plots by the log of the price.

Choropleth Maps¶

A Choropleth Map is a map composed of colored polygons, showing the variable of interest at different color depth across geographies.Plotly express has a function called px.choropleth that be used to plot Choropleth Map. The challenges here are getting the geometry information. There are two ways, one is to use the built-in geometries in plotly when plot US States (use the state name directly) and world countries (use ISP-3 code). Another way is to look for GeoJSON files where each location has geometry information. In the following example, we will show the plot in US States with a synthetic data set.

Histogram¶

A histogram is a graph that shows the frequency distribution of numerical data. In addition to helping us understand frequency, histograms are also useful for detecting outliers. We can use the px.histogram() function from Plotly to draw histograms for specific columns, as long as the data type is numerical. Let's check the following example:

Practice

Check the "surface_covered_in_m2" Histogram.

Boxplots¶

A boxplot is a graph that shows the minimum, first quartile, median, third quartile, and the maximum values in a dataset. Boxplots are useful because they provide a visual summary of the data, enabling researchers to quickly identify mean values, the dispersion of the data set, and signs of skewness. In the following example, we will explore how to draw boxplots for specific columns of a DataFrame.

Check the boxplot for column "price":

If you want to check the distribution of a column value by different categories, defined by another categorical column, you can add an x argument to specify the name of the categorical column. In the following example, we check the price distribution across different property types:

Practice

Check the "surface_covered_in_m2" distribution by property types.

Bar Chart¶

A bar chart is a graph that shows all the values of a categorical variable in a dataset. They consist of an axis and a series of labeled horizontal or vertical bars. The bars depict frequencies of different values of a variable or simply the different values themselves. The numbers on the y-axis of a vertical bar chart or the x-axis of a horizontal bar chart are called the scale.

In the following example, we will see some bar plots based on the Mexico City real estate dataset. Specifically, we will count the number of observations in each borough and plot them. We first need to read the data set and extract Borough and other location information from column "place_with_parent_names".

We can calculate the number of real estate showing in the data set by Borough using value_counts(), then plot it as bar plot:

We can plot more expressive bar plots by adding more arguments. For example, we can plot the number of observations by borough and property type. First of all, we need use groupby to calculate the aggregated counts for each Borough and property type combination:

By specifying x, y and color, the following bar graph shows the total counts by Borough, with different property types showing in different colors. Note y has to be numerical, while x and color are usually categorical variables.WQU WorldQuant University Applied Data Science Lab QQQQ

Note the argument barmode is specified as 'relative', which is also the default value. In this mode, bars are stacked above each other. We can also use 'overlay' where bars are drawn on top of each other.

If we want bars to be placed beside each other, we can specify barmode as "group":

Practice

Plot bar plot for the number of observations by property types in "mexico-city-real-estate-1.csv".

References and Further Reading¶

• Official plotly express Documentation on Scatter Plots
• Official plotly Express Documentation on 3D Plots
• Official plotly Documentation on Notebooks
• plotly Community Forum Post on Axis Labeling
• plotly express Official Documentation on Tile Maps
• plotly Choropleth Maps in Python Document
• plotly express Official Documentation on Figure Display
• Online Tutorial on String Conversion in Pandas
• Official Pandas Documentation on using Lambda Functions on a Column
• Official Seaborn Documentation on Generating a Heatmap
• Online Tutorial on Correlation Matrices in Pandas
• Official Pandas Documentation on Correlation Matrices
• Official Matplotlib Documentation on Colormaps
• Official Pandas Documentation on Box Plots
• Online Tutorial on Box Plots
• Online Tutorial on Axes Labels in Seaborn and Matplotlib
• Matplotlib Gallery Example of an Annotated Heatmap

Copyright 2022 WorldQuant University. This content is licensed solely for personal use. Redistribution or publication of this material is strictly prohibited.

There are many ways to interact with data, and one of the most powerful modes of interaction is through visualizations. Visualizations show data graphically, and are useful for exploring, analyzing, and presenting datasets. We use four libraries for making visualizations: pandas, Matplotlib, plotly express, and seaborn. In this section, we'll focus on using seaborn.

Scatter Plots¶

A scatter plot is a graph that uses dots to represent values for two different numeric variables. The position of each dot on the horizontal and vertical axis indicates values for an individual data point. Scatter plots are used to observe relationships between variables, and are especially useful if you're looking for correlations.

In the following example, we will see some scatter plots based on the Mexico City real estate data. Specifically, we can use scatter plot to show how "price" and "surface_covered_in_m2" are correlated. First we need to read the data set and do a little cleaning.

Use seaborn to plot the scatter plot for "price" and "surface_covered_in_m2":

There is a very useful argument in scatterplot called hue. By specifying a categorical column as hue, seaborn can create a scatter plot between two variables in different categories with different colors. Let's check the following example using "property_type":

Practice

Plot a scatter plot for "price" and "surface_total_in_m2" by "property_type" for "mexico-city-real-estate-1.csv":

Bar Charts¶

A bar chart is a graph that shows all the values of a categorical variable in a dataset. They consist of an axis and a series of labeled horizontal or vertical bars. The bars depict frequencies of different values of a variable or simply the different values themselves. The numbers on the y-axis of a vertical bar chart or the x-axis of a horizontal bar chart are called the scale.

In the following example, we will see some bar plots based on the Mexico City real estate dataset. Specifically, we will count the number of observations in each borough and plot them. We first need to import the dataset and extract the borough and other location information from column "place_with_parent_names".

Let's check the example of a bar plot showing the value counts of each borough in the dataset. We first need to create a DataFrame showing the value counts:

Since there are 16 different categories in Borough, we should increase the default plot size and rotate the x axis to make the plot more readable using the following syntax:

Practice

Plot a bar plot showing the value counts for property types in "mexico-city-real-estate-1.csv":

Correlation Heatmaps¶

A correlation heatmap shows the relative strength of correlations between the variables in a dataset. Here's what the code looks like:

Notice that we dropped the columns and rows with missing entries before plotting the graph.

This heatmap is showing us what we might already have suspected: the price is moderately positively correlated with the size of the properties.

Practice

The seaborn documentation on heat maps indicates how to add numeric labels to each cell and how to use a different colormap. Modify the plot to use the viridis colormap, have a linewidth of 0.5 between each cell and have numeric labels for each cell.

References and Further Reading¶

• Official Plotly Express Documentation on Scatter Plots
• Official Plotly Express Documentation on 3D Plots
• Official Plotly Documentation on Notebooks
• Plotly Community Forum Post on Axis Labeling
• Plotly Express Official Documentation on Tile Maps
• Plotly Express Official Documentation on Figure Display
• Online Tutorial on String Conversion in Pandas
• Official Pandas Documentation on using Lambda Functions on a Column
• Official seaborn Documentation on Generating a Heatmap
• Online Tutorial on Correlation Matrices in Pandas
• Official Pandas Documentation on Correlation Matrices
• Official Matplotlib Documentation on Colormaps
• Official Pandas Documentation on Box Plots
• Online Tutorial on Box Plots
• Online Tutorial on Axes Labels in seaborn and Matplotlib
• Matplotlib Gallery Example of an Annotated Heatmap

Copyright 2022 WorldQuant University. This content is licensed solely for personal use. Redistribution or publication of this material is strictly prohibited. WQU WorldQuant University Applied Data Science Lab QQQQ

Copyright 2022 WorldQuant University. This content is licensed solely for personal use. Redistribution or publication of this material is strictly prohibited.

Databases: SQL

Working with SQL Databases¶

A database is a collection of interrelated data. The primary goal of a database is to store and retrieve information in a convenient and efficient way. There are many types of databases. In this section, we will be dealing with a relational database. A relational database is a widely used database model that consists of a collection of uniquely named tables used to store information. The structure of a database model with its tables, constraints, and relationships is called a schema.

A Structured Query Language (SQL), is used to retrieve information from a relational database. SQL is one of the most commonly used database languages. It allows data stored in a relational database to be queried, modified, and manipulated easily with basic commands. SQL powers database engines like MySQL, SQL Server, SQLite, and PostgreSQL. The examples and projects in this course will use SQLite.

A table refers to a collection of rows and columns in a relational database. When reading data into a pandas DataFrame, an index can be defined, which acts as the label for every row in the DataFrame.

Connecting to a Database¶

ipython-sql¶

Magic Commands¶

Jupyter notebooks can run code that is not valid Python code but still affect the notebook . These special commands are called magic commands. Magic commands can have a range of properties. Some commonly used magic functions are below:

We will be leveraging magic commands to work with a SQLite database.

ipython-sql¶

ipython-sql allows you to write SQL code directly in a Jupyter Notebook. The %sql (or %%sql) magic command is added to the beginning of a code block and then SQL code can be written.